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I .  Introduction 

Programmers  have  historically  used  testing  to  convince 
themselves  and  others  that  their  software  works.  Although 
testing  has  limitations,  as  do  other  reliability  methods,  it 
has  one  major  advantage:  it  is  the  only  method  in  which  the 
actual  behavior  of  the  software  can  be  observed.  Thus, 
errors  or  oversights  in  the  supporting  environment, 
including  the  available  validation  tools,  translators, 
operating  system,  and  hardware,  may  be  detected.  Moreover, 
the  actual  performance  can  be  evaluated  for  efficiency  and 
usefulness . 


We  have  been  using  the  ATTEST  system  [CLA78a,  CLA78b] 
as  a  vehicle  to  design,  implement,  and  experiment  with 
various  aspects  of  symbolic  execution  and  software  testing. 
ATTEST  is  an  experimental  symbolic  execution  system  that 
attempts  to  generate  test  data  to  satisfy  a  variety  of 
testing  criteria.  It  is  composed  of  three  major  components: 
path  selection,  symbolic  execution,  and  test  data 
generation.  The  path  selection  component  is  concerned  with 
selecting  program  paths  that  satisfy  user  selected  testing 
criteria.  Usually  this  involves  choosing  paths  that  contain 
untested  segments  of  code.  The  symbolic  execution  component 
then  analyzes  each  selected  path.  During  symbolic 
execution,  the  computations  on  a  path  are  represented  as 
algebraic  expressions,  the  domain  for  the  input  values  is 
defined  by  a  set  of  constraints,  and  error  detection  is 
done.  The  test  data  generation  component  checks  the  domain 
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test  data  that  would  drive  execution  down  the  selected  path. 

ATTEST  is  designed  so  that  it  can  either  augment 
user-selected  test  data  or  independently  generate  sets  of 
test  data.  Although  ATTEST  will  not  guarantee  correctness 
of  a  program,  it  does  surpass  current  testing  techniques  in 
that  it  offers  a  systematic  method  of  selecting  test  data  to 
achieve  a  determined  measure  of  program  coverage,  and  it 
provides  symbolic  representations  of  a  path’s  computation 
and  domain  that  can  be  utilized  by  a  variety  of  testing 
strategies. 

During  the  grant  period,  we  addressed  several  issues 
relating  to  the  development  of  a  testing  system.  In  the 
sections  that  follow,  an  overview  of  our  investigation  in 
each  of  these  issues  is  provided. 

II.  Path  Selection 

A  well  accepted  criterion  for  testing  programs  is  to 
execute  al 1  branches  at  least  once.  Although  this  criterion 
does  not  guarantee  detection  of  all  errors,  it  does 
guarantee  a  minimal  level  of  program  coverage.  Manually 
creating  test  data  to  satisfy  this  criterion  can  be  very 
tedious  and,  at  times,  very  difficult.  Experiments  have 
shown  that  without  automated  assistance,  this  level  of 
testing  is  rarely  achieved  [STU73].  Furthermore,  the 
sections  of  code  that  are  the  most  tedious  to  test, 
typically  code  for  checking  the  validity  of  parameters  and 
input  data,  are  usually  the  most  susceptible  to  this 
approach,  thus  freeing  the  programmer  to  concentrate  on  more 
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rigorous  testing  strategies  for  the  more  abstruse  code 
segments . 

Previously  attempted  solutions  to  the  problem  of 
generating  a  set  of  paths  to  execute  all  branches  have  been 
unsuccessful  or  inefficient.  Methods  considering  only 
program  structure  [PAI75,  McC76]  have  failed  because  of  the 
large  number  of  nonexecutable  paths  that  they  generate. 
Some  attempts  had  been  made  to  detect  incompatible  branch 
conditions  [KRA73,  OST77],  however,  the  complexity  of 
incorporating  this  information  into  a  static  path  selection 
algorithm  is  NP  complete  [GAB76].  The  SMOTL  system  [BIC793 
statically  describes  the  complete  set  of  paths  for  executing 
all  branches,  but  then  symbolically  evaluates  each  path  to 
eliminate  the  nonexecutable  ones;  this  method  works,  but  is 
extremely  inefficient. 

We  developed  an  efficient,  dynamic  method  for  selecting 
paths.  Instead  of  describing  all  the  paths  before  symbolic 
execution  is  initiated,  this  method  selects  paths  during  the 
symbolic  execution  process.  This  allows  information 
obtained  about  a  portion  of  a  path  to  be  utilized  when 
selecting  the  rest  of  the  path.  In  addition,  this  method 
maintains  a  path  history  of  the  program  so  that  it  can 
"learn"  from  past  experience.  When  a  conditional  statement 
is  encountered  on  a  path,  the  current  path  status  and  path 
history  are  used  to  determine  which  branch  from  this 
conditional  statement  should  be  selected  next.  The  branch 
that  appears  to  have  the  highest  potential  for  achieving 
additional  program  coverage  is  selected. 
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This  path  selection  method  is  not  just  applicable  to 
the  all  branches  testing  criterion,  but  is  applicable  to  a 
range  of  criteria  based  on  program  coverage.  Our  program 
coverage  method  of  path  selection  requires  that  the  path 
selection  criteria  be  defined  by  the  ATTEST  user  by 
specifying,  among  other  things,  a  relative  weight  to  be 
given  for  branch  coverage,  for  statement  coverage,  and  for 
exercising  loop  boundary  conditions. 

In  an  experiment  using  the  program  coverage  method 
[W0080],  the  ATTEST  system  performed  efficiently  and 
achieved  or  approached  the  specified  testing  criteria  for 
all  programs  that  were  attempted.  For  example,  when  a  high 
relative  weight  was  selected  for  the  all  branches  testing 
criterion,  at  least  ninety  percent  of  the  branches  were 
exercised  in  all  the  programs  that  were  analyzed.  In 
addition,  only  the  minimum  number  of  paths  was  usually 
required  to  satisfy  the  selected  testing  criterion. 

Although  the  above  method  has  had  promising  results 
when  testing  programs,  more  comprehensive  testing  may 
sometimes  be  desired.  Thus,  we  also  developed  and 
investigated  an  approach  that  approximates  testing  all  the 
paths  in  a  procedure  [MAR79  ] .  The  problem  with  an  al 1  paths 
testing  criterion  is  that  loops  within  a  procedure  may 
result  in  an  infinite  number  of  paths.  Thus,  we  devised  an 
algorithm  for  selecting  a  subset  of  the  paths  through  a  loop 
that  we  believe  would  increase  the  likelihood  of  detecting 
loop  boundary  errors.  Our  observations  about  the  best  such 
subset  of  paths  to  approximate  all  paths  testing  agrees 
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closely  with  the  subset  described  in  EHOW79].  Except  when 
selecting  paths  in  a  loop,  this  algorithm  for  path  selection 
employs  a  depth  first  search  algorithm  and  uses  many  of  the 
same  techniques  that  proved  successful  in  the  coverage 
method  described  above. 

III.  Ar ray  Element  Determination 

During  symbolic  execution  it  is  difficult  to  determine 
the  value  of  an  array  index  that  depends  on  input  values. 
Although  the  occurrence  of  such  an  indeterminate  array  index 
can  be  represented  symbolically  for  display  to  the  user, 
such  an  occurrence  usually  precludes  any  further  error 
detection  analysis,  consistency  determination  of  the  path 
domain,  or  test  data  generation.  In  our  experience,  this 
problem  has  been  the  major  drawback  to  symbolic  execution 
and  has  severely  limited  the  types  of  programs  that  can  be 
analyzed.  Solutions  to  finding  suitable  indexes  have  been 
proposed  [ RAM76  ] ,  but  they  are  not  always  applicable  and  are 
extremely  inefficient,  often  requiring  the  system  to  backup 
to  a  previous  statement  on  the  path. 

We  have  devised  a  method  for  handling  indeterminate 
array  indexes  [M0079]  that  seems  superior  to  previously 
attempted  solutions  to  this  problem.  This  method  represents 
all  indeterminate  array  indexes  symbolically,  as  well  as 
symbolically  representing  all  information  that  subsequently 
depends  on  these  indexes.  These  representations,  although 
too  complicated  to  be  meaningful  to  the  user,  are  internal 
representations  that  allow  analysis  to  continue.  Using 
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these  representations,  the  inequality  solver  in  the  test 
data  generation  component  determines  a  legal  range  of  values 
for  each  indeterminate  index.  It  then  determines  if  an 
index  in  this  range  can  be  found  so  that  the  index  and  its 
corresponding  array  element  value  result  in  an  executable 
path.  If  such  an  index  is  found,  symbolic  execution 
continues.  At  some  later  point  during  symbolic  execution 
the  selected  array  index  may  no  longer  be  appropriate;  the 
path  is  nonexecutable  unless  other  indexes  satisfying  the 
current  set  of  path  constraints  can  be  found.  Our  array 
method  employs  an  efficient  algorithm  for  determining  the 
indexes  that  must  be  modified  and  for  selecting  alternative 
values  if  any  exist.  Moreover,  the  system  does  not  have  to 
backup;  it  must  only  modify  the  symbolic  representations  of 
the  effected  indexes  to  correspond  to  their  new  values. 

We  are  cautiously  optimistic  about  this  algorithm.  It 
takes  into  account  the  necessary  relationships  between  array 
indexes  as  well  as  the  relationship  between  each  selected 
index  and  the  value  of  its  corresponding  array  element.  In 
the  worst  case,  the  algorithm  degrades  to  enumeration  over 
all  possible  array  indexes,  but  this  appears  to  be  a  rare 
occurrence.  We  have  manually  evaluated  this  algorithm  for 
several  programs  and  in  all  cases  the  algorithm  quickly 
selected  appropriate  indexes  or  determined  that  the  paths 
were  nonexecutable. 


Page  8 


IV.  Test  Environment 

We  completed  an  initial  design  for  an  ATTEST  Interface 
Description  language  (AID)  [WIN78].  In  this  design  we 
addressed  two  questions:  what  capabilities  are  needed  in  a 
symbolic  execution  session;  and  how  can  these  requests  be 
easily  communicated.  The  first  question  led  to  a  command 
language  that  tries  to  capture  the  testing  environment.  The 
second  question  led  to  considerable  care  being  taken  in 
designing  a  natural,  uniform,  and  yet  concise  syntax  for  the 
language. 

It  is  our  belief  that  program  testing  should  be 
considered  throughout  software  development  and  not  just  as 
an  afterthought  to  the  implementation  phase.  Therefore,  the 
AID  language  supports  a  facility  for  describing  a  test 
harness  that  is  useful  during  the  design  and  implementation 
phases  of  software  development  [OGD78].  Usually  the  driver 
program  and  program  stubs  in  a  test  harness  are  simple, 
static  procedures  that  limit  realistic  program  testing.  AID 
supports  a  facility  for  symbolic  and  dynamic  descriptions  of 
these  procedures.  This  capability  should  help  simulate  the 
actual  environment  more  closely  and  thus  allow  more 
realistic  testing  during  software  development.  Although 
test  data  generation  is  usually  not  possible  during  the 
design  phase,  symbolic  execution  resulting  in  a  symbolic 
representation  of  the  design  can  be  done. 
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To  provide  support  during  the  design  phase,  the  AID 
commands  and  the  programming  language  statements  (in  our 
case  FORTRAN)  can  be  intermixed.  Initially  a  procedure  can 
be  described  predominately  by  AID  commands.  As  the  design 
proceeds,  additional  AID  commands  may  be  added  to  refine  the 
design.  Gradually  the  AID  commands  may  be  replaced  by 
FORTRAN  code.  Thus,  AID  facilitates  program  design  by 
stepwise  refinement. 

A  study  of  the  use  of  AID  during  both  the  design  and 
testing  phases  was  done  [WIN79].  This  study  revealed 
several  problems  with  the  initial  design,  the  most  notable 
being  the  lack  of  a  facility  to  associate  AID  commands  with 
only  the  paths  related  to  a  particular  path  selection 
criterion.  For  the  most  part,  however,  AID  was  easy  to  use 
and  provided  the  desired  interface.  Although  only  the  most 
rudimentary  features  of  AID  have  been  implemented  in  ATTEST, 
the  MUST  project  [TAY793  is  building  a  symbolic  execution 
system  that  provides  an  interface  based  on  AID. 

In  a  related  effort,  we  evaluated  three  techniques  for 
handling  procedure  calls  in  a  testing  environment.  The 
first  and  most  straightforward  technique  involves 
symbolically  executing  a  path  in  the  called  procedure  at  the 
time  the  procedure  is  invoked.  This  resembles  normal 
execution  and  requires  that  all  the  called  procedures  be 
available  for  symbolic  execution.  The  second  technique, 
called  procedure  substitution,  saves  the  results  from 
symbolically  executing  a  procedure  so  that  when  this 
procedure  is  subsequently  invoked  by  another  procedure,  the 
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saved  results  can  be  ’substituted'  for  the  call.  This 
technique  is  supportive  of  bottom-up  integration  and  testing 
methods  in  which  low  level  procedures  are  developed  and 
tested  before  higher  level  procedures.  An  efficient 
implementation  of  this  technique  poses  many  problems.  As 
our  investigation  showed  [W0080],  it  is  frequently  the  case 
that  it  is  more  efficient  to  symbolically  execute  a 
procedure  than  to  substitute  the  results  from  a  previous 
execution.  Fortunately,  an  accurate  prediction  can  usually 
be  made  about  which  technique  would  be  more  efficient  to  use 
for  a  particular  procedure  call.  The  third  technique 
requires  that  the  user,  employing  AID  commands,  describe  the 
desired  effect  of  an  invoked  procedure.  This  technique 
supports  top  down  software  development  where  high  level 
procedures  are  implemented  and  tested  before  lower  level 
procedures.  AID  provides  the  user  with  a  wide  range  of 
capabilities  for  describing  the  effects  of  a  procedure  call. 
The  descriptions  usually  provided  by  a  user,  however,  are 
simpler  to  manage  than  the  descriptions  automatically 
provided  by  procedure  substitution,  since  the  user  only 
describes  the  pertinent  testing  information.  Thus,  this 
third  technique  can  be  efficiently  implemented. 


V.  Format  Analysis 

Since  automatic  testing  systems  are  concerned  with 
generating  legal  input  data,  it  is  important  to  examine  the 


corresponding  format  specifications  that  exist  in  languages 


like  FORTRAN  and  PL/1.  Although  symbolic  execution  systems 


have  been  developed  for  these  two  languages,  none  of  these 
systems  have  taken  format  specifications  into  account. 


We  developed  an  algorithm  to  detect  data  list-format 
list  correspondence  during  compilation  [ABR793.  ThiJ 
algorithm  is  efficient  in  that  it  retains  much  of  the 
original  structure  (e.g.,  iteration  counts  and  nesting)  of 
the  initial  format  specification.  A  study  of  250  I/O 
statements  found  that  9^%  of  the  data-list,  format-list 
pairs  could  be  completely  analyzed  at  compile  time  by  our 
algorithm. 

Compiler  optimization  is  another  area  where  this 
algorithm  is  being  successfully  applied.  Knuth's  early 
study  of  FORTRAN  programs  [KNU713  found  that  twenty-five 
percent  of  the  execution  time  was  spent  in  I/O  editing,  so 
it  is  not  suprising  that  this  algorithm  has  been  implemented 
and  is  new  being  commercially  employed  [MCA80]  as  an 
optimization  technique. 


VI.  Employing  Specifications  in  the  Testing  Process 
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the  implementation.  Such  methods  are  unlikely  to  detect 
errors  that  arise  when  an  implementation  neglects  aspects  of 
the  problem,  whereas  utilizing  an  understanding  of  the 
specification  may  direct  attention  to  such  errors. 
Recently,  several  attempts  have  been  made  to  employ  sources 
of  information  over  and  above  the  implementation  in 


selecting  test  data.  Goodenough  and  Gerhart  [G0076]  have 
argued  that  the  specification  and  the  implementation  are 
both  valuable  sources  of  information  that  must  be  used  by 
testing  methods.  Thus  far,  only  a  few  methods  [GEL78, 
WEY80]  have  been  developed  to  exploit  formal  specifications, 
even  though  such  specifications  are  becoming  more  readily 
available  as  their  value  in  the  development  of  reliable 
software  is  recognized. 

We  have  developed  a  method  [RIC78b,  79,  81],  called 
partition  analysis ,  that  assists  in  program  testing  and 
program  verification  by  i ncorporating  information  from  both 
a  formal  specification  and  an  implementation  for  a 
procedure.  The  partition  analysis  method  employs  symbolic 
execution  techniques  to  partition  the  set  of  input  data  into 
procedure  subdomains ,  so  that  the  elements  of  each  subdomain 
are  treated  uniformly  by  the  specification  and  processed 
uniformly  by  the  implementation.  By  forming  these 
subdomains,  the  procedure  domain  is  divided  into  more 
manageable  units,  as  is  the  task  of  demonstr ating  program 
reliability.  Information  related  to  each  procedure 
subdomain  is  used  to  guide  in  the  selection  of  test  data 
that  reveals  errors  in  the  implementation  or  provides 
confidence  in  its  correctness.  This  information  is  also 
used  to  verify  consistency  between  the  specification  and  the 
implementation.  Moreover,  the  test  data  selection  process, 
called  partition  analysis  testing ,  and  the  verification 
process,  called  partition  analysis  verification,  are  used  to 


enhance  each  other;  the  execution  of  some  elements  in  the 
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subdomain  may  assist  in  verification,  while  the  verification 
process  may  direct  the  selection  of  test  data. 

Our  investigation  of  partition  analysis  led  to  some 
related  research  endeavors.  To  create  a  finite  number  of 
subdomains,  loop  analysis  techniques  [CHE79]  were  evaluated 
and  expanded  [CLA8lb,  81c,  8 1 d ] .  Work  on  determining  test 
data  for  subdomains  led  to  substantial  improvements  in  the 
Domain  Testing  Strategy  [HAS80]  originally  proposed  by  White 
and  Cohen  [WHI80].  Finally,  in  evaluating  appropriate 
specification  and  design  languages  for  use  with  partition 
analysis,  recommendations  for  representing  data  abstraction 
and  modularity  [CLA80]  have  been  proposed  as  well  as  more 
general  recommendations  for  environments  supporting  software 
development  activities  [CLA81a]. 

VII.  Implementation  Status 

A  portion  of  our  time  was  spent  in  implementing  and 
evaluating  many  of  the  test  data  generation  and  symbolic 
execution  features  we  were  investigating.  During  the  grant 
period  our  major  implementation  efforts  included  the 
following: 

-  Three  methods  of  path  selection  [MAR78,  W0080]. 

-  An  efficient  system  for  simplifying  constraints 
[RIC78a] . 

-  An  improved  interface  to  the  inequality  solver  that 
recognizes  redundancies  and  dominance  relationships 
[DIL81  ]. 


Some  of  the  features  in  the  AID  language  [WIN78]. 


Page  14 


ATTEST  is  predominately  written  in  FORTRAN  4  and  5.  It 

is  designed  to  be  an  experimental  system.  Storage  and 

execution  time  considerations  have  always  been  given  a  lower 

priority  than  flexibility  and  adaptability  considerations. 

Originally  ATTEST  was  implemented  on  the  C DC  Cyber  but  was 

0 

successfully  moved  to  our  recently  acquired  department 
research  facility.  ATTEST  now  runs  on  the  VAX  under  VMS. 
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